PYTHON ROADMAP¶

1. PYTHON CODING SKILLS

BASIC SKILLS

  • Variables, numbers, strings,lists.
  • Dictionary, tuple
  • If, for control blocks
  • Functions
  • Read, write files
  • Modules

INTERMEDIATE SKILLS

  • Exception Handling
  • Classes, Objects
  • Inheritance
  • Iterators
  • Generators
  • List/dict comprehensions
  • Sets, command line argparse

ADVANCED SKILLS

  • Multithreading
  • Multiprocessing
  • Multiprocessing lock & pool
  • Unit tests : pytest
  • Decorators
  • Context Managers

2. DATA STRUCTURES & ALGORITHMS

3. DEBUGGING SKILLS

4. USING LIBRARIES WISELY

5. PEP8 & CODING FAST PRACTICES

6. OPEN SOURCE CONTRIBUTIONS & CODE REVIEWS

7. MASTER THE ART OF ASKING FOR HELP

8. PROJECT, PRACTICE, REPEAT UH

WHY PYTHON?¶

  • Very easy to learn
  • Code development speed
  • No compilation
  • Library for everything
  • Used heavily in, Data Science, Machine Learning, Scientific Computing
  • Plenty of job oppurtunities

1. VARIABLES IN PYTHON¶

Its a container that can hold any data

In [1]:
'''Example problem: Find out total monthly expense given individual expense items
   Sol: Store in-user expenses in some place and then use that place to do a sum of 
   all those individual expenses to come up with total expense'''

rent = 1220 # '=' is a assignment operator, left side is variable and right is the value stored in the variable
gas = 202.5
groceries = 305.6

total = rent + gas + groceries
print(total) 

1728.1

2. NUMBERS IN PYTHON¶

We can most likely do any mathematical operations [like a calculator]

In [2]:
5 + 2
Out[2]:
7

We can also do operations on stored variables

In [3]:
a = 5 
b = 6
c = a * b
print(c)

30

We can use round() to control the decimal points

In [4]:
a = 10 
b = 3
c = a / b
print(c)

3.3333333333333335
In [5]:
round(c, 2)
Out[5]:
3.33
In [6]:
round(c, 3)
Out[6]:
3.333

3. STRINGS IN PYTHON¶

Its used to store text data

In [7]:
text = 'ice cream'
print(text)

ice cream
In [8]:
'''It stores it in a sequence of characters 
   0 1 2 3 4 5 6 7 8 
   i c e   c r e a m
If I want get the character on particular index I can do it like this one on below
'''
text[0]
Out[8]:
'i'
In [9]:
'''I can also give a range where we technically call it as slicing in python [start : stop]'''

text[0:3]
Out[9]:
'ice'
In [10]:
text[4:9]
Out[10]:
'cream'

4. LISTS IN PYTHON¶

We can store n number of elements in form of an array

In [11]:
'''Lets take a scenerio that we go to a grocery store and I need some items like
   bread, nutella, maybe some crackers like hide n seek, or fruits, I can store all 
   this in one place in python using lists'''

items = ["bread", "nutella", "hide n seek", "fruits"]
In [12]:
items
Out[12]:
['bread', 'nutella', 'hide n seek', 'fruits']
In [13]:
'''The way this list stored is like
    0        1         2              3
    bread    nutella   hide n seek    fruits
It is similar to strings, the way it stored on memory locations
'''
items[0]
Out[13]:
'bread'
In [14]:
items[2]
Out[14]:
'hide n seek'
In [15]:
'''If I need chips instead of bread I can modify like this'''
items[0] = 'chips'
In [16]:
items
Out[16]:
['chips', 'nutella', 'hide n seek', 'fruits']
In [17]:
'''To access range of elements'''
items[0:2]
Out[17]:
['chips', 'nutella']
In [18]:
items[-1] # negative index '-1' means first index seen from the end
Out[18]:
'fruits'
In [19]:
# We can use append() to add an element
items.append("paneer")
In [20]:
items
Out[20]:
['chips', 'nutella', 'hide n seek', 'fruits', 'paneer']
In [21]:
'''We can also append on specific locations'''
items.insert(1, '7-up')
In [22]:
items
Out[22]:
['chips', '7-up', 'nutella', 'hide n seek', 'fruits', 'paneer']
In [23]:
'''We can also join two lists'''
items_cleaning = ['harpic', 'lizol']
In [24]:
complete_items = items + items_cleaning
In [25]:
complete_items
Out[25]:
['chips',
 '7-up',
 'nutella',
 'hide n seek',
 'fruits',
 'paneer',
 'harpic',
 'lizol']
In [26]:
'''HEADS UP: While concatenating you can add a string directly to the list, it will throw an error'''
Out[26]:
'HEADS UP: While concatenating you can add a string directly to the list, it will throw an error'
In [27]:
# We can use len() to check the length of list
len(complete_items)
Out[27]:
8
In [28]:
'''There should be a better way than reading the elements of the list one by one
   The better way to do this is "in" operator'''
"chips" in items
Out[28]:
True
In [29]:
"coke" in items
Out[29]:
False

5. IF STATEMENTS IN PYTHON¶

SAMPLE PROGRAM 1 : Write a program that asks user to enter a number. Program should then tell if number is odd or even

In [30]:
num = input("Enter a number: ") # Whatever the input is, it initially stored as a string, so we have to convert it to appropriate variable

Enter a number: 3
In [31]:
num = int(num)
In [32]:
if num % 2 == 0:
    print("number is even")
else:
    print("number is odd")

number is odd

SAMPLE PROGRAM 2 : Write a program that asks user to enter dish name and it should print which cuisine is that dish

In [33]:
indian = ["sambar", "curd", "parotha"]
chinese = ["noodles", "fried rice", "dim sum"]
italian = ["pasta", "pizza", "risotto"]
In [34]:
dish = input("Enter a dish name: ")

Enter a dish name: noodles
In [35]:
if dish in indian:
    print("Indian cuisine")
elif dish in chinese:
    print("Chinese cuisine")
elif dish in italian:
    print("Italian cuisine")
else:
    print("We dont sell it here -_-")

Chinese cuisine

6. FOR STATEMENTS IN PYTHON¶

SAMPLE PROBLEM : Store monthly expenses in a list and find out total expenses for all months

In [36]:
# traditional way without using for loop
exp = [23400, 25000, 21000, 31000, 29800]
total = exp[0] + exp[1] + exp[2] + exp[3] + exp[4]
print(total)
'''The problem with this way is its a freaking long code where my brain will implode when it comes to like 100 elements'''

130200
In [37]:
# The better way is to use for loop
total = 0 # We keep it as 0 since its gonna get added when it runs on loop
for item in exp:
    total = total + item
print(total)
# If you look at the difference in code, this is far more better and brain wont implode ffs

130200
In [38]:
'''What if we want print numbers using range() function "range(start, stop)"'''

for i in range(1,11):
    print(i)

1
2
3
4
5
6
7
8
9
10

In our monthly expense example, print month number and expense and then in the end print total expense

In [41]:
total = 0
for i in range(len(exp)):
    print('Month:',(i+1), ' Expense:',exp[i]) # Giving 'i+1' because in range funtion we will only specify end index, if start index is not specified, it will be 0 by default, I want to start from 1
    total = total + exp[i]
    
print('Total expense is:',total)

Month: 1  Expense: 23400
Month: 2  Expense: 25000
Month: 3  Expense: 21000
Month: 4  Expense: 31000
Month: 5  Expense: 29800
Total expense is: 130200
In [42]:
'''If my program goal is achieved, the for loop should be terminated
   A better example, like if we lost our car key in home, we begin searching it,
   We search in multiple places and once found we stop searching. Lets demonstrate'''

key_location = "chair"
locations = ["shed", "pooja room", "living room", "chair", "backyard", "closet"]
for i in locations:
    if i == key_location:
        print("key is found in", i)
        break # terminate
    else:
        print("key is not found in",i)

key is not found in shed
key is not found in pooja room
key is not found in living room
key is found in chair
In [43]:
'''Another useful statement is "continue", its merely the skipping iterations.
   Lets take a problem, Print square of numbers between 1 to 5 except even numbers'''
for i in range(1,6):
    if i % 2 == 0:
        continue # if it wrong or right it doesnt care, it just print what satisfies the condition and continue running the loop by skipping the wrong ones
    print(i*i)

1
9
25
In [44]:
# USING WHILE LOOP
i = 1
while i<=5:
    print(i)
    i = i + 1

1
2
3
4
5

7. FUNCTIONS IN PYTHON¶

Functions is a block of code that performs a specific task

In [45]:
'''Lets take washing machine as real time example, we put dirty clothes in the machine, it will wash it and give out the 
   clean clothes, See it in a format, Dirty clothes here acts as funtion arguments, And we do some tasks like
   Function : wash_clothes
   1. Add water/detergent
   2. Wash clothes
   3. Give out clean clothes
   So the output here is clean clothes[return value]
'''
'''
SAMPLE PROBLEM : YOU ARE GIVEN TWO LISTS OF NUMBERS AND YOU NEED TO FIND
                 TOTAL OF EACH OF THESE LIST
'''
# traditional way
raj_exp_list = [21000, 34000, 35000]
rita_exp_list = [2000, 5000, 7000]

total = 0
for item in raj_exp_list:
    total = total + item
print("Raj's total expenses:",total)

total = 0
for item in rita_exp_list:
    total = total + item
print("Rita's total expenses:",total)
'''The problem with this code is we are repeating three lines of code again and again, lets see we need 
   to find the total expense of 10 people, then it becomes tedious right? So thats where functions are used'''

Raj's total expenses: 90000
Rita's total expenses: 14000
In [54]:
# Using the function
def calculate_total(exp): # def is the special keyword that tells python that we are writing function
    total = 0
    for item in exp:
        total = total + item
    return total

raj_total = calculate_total(raj_exp_list)
rita_total = calculate_total(rita_exp_list)

print("Raj's total expenses:",raj_total)
print("Rita's total expenses:",rita_total)

Raj's total expenses: 90000
Rita's total expenses: 14000
In [53]:
# Another example, sum of two numbers
total2 = 0 # global variable
def sum(a,b): # defining function
    total1 = a + b # local variable
    print("Total from inside the function:",total1)
    return total # return value

n = sum(5,6) # calling function
print("Total from outside the function:",total2)

Total from inside the function: 11
Total from outside the function: 0

8. DICTIONARIES IN PYTHON¶

Dictionary allows us to store key, value pairs. It also called as Maps, Hashtables, Associate Arrays. A best real time example for dictionaries are telephone directory. Let's have a look,

In [55]:
d = {"raju":9876514590, "ranga":8976542310, "ravi":6309176532}
d
Out[55]:
{'raju': 9876514590, 'ranga': 8976542310, 'ravi': 6309176532}
In [56]:
d["ranga"] # We can access specific key values by calling that key
Out[56]:
8976542310
In [57]:
# Adding new entry in a dictionary
d["sita"] = 8797165342
d
Out[57]:
{'raju': 9876514590,
 'ranga': 8976542310,
 'ravi': 6309176532,
 'sita': 8797165342}
In [58]:
# Deleting a entry in a dictionary
del d["ranga"]
d
Out[58]:
{'raju': 9876514590, 'ravi': 6309176532, 'sita': 8797165342}
In [59]:
for key in d:
    print("key:",key,"value:",d[key])

key: raju value: 9876514590
key: ravi value: 6309176532
key: sita value: 8797165342
In [61]:
for k,v in d.items():
    print("key:",k,"value:",v)

key: raju value: 9876514590
key: ravi value: 6309176532
key: sita value: 8797165342
In [62]:
"ravi" in d
Out[62]:
True
In [63]:
"ranga" in d
Out[63]:
False
In [64]:
d.clear() # clear every entries in a dictionary
d
Out[64]:
{}

9. TUPLES IN PYTHON¶

Tuple is a list of values grouped together

In [65]:
#For example, if we want represent geometric points in 2D plane, we will use tuple
point = (5,9)
In [66]:
point[0]
Out[66]:
5
In [67]:
'''
   It is more likely similar to list, but it has major difference.
   In List, all values have similar meaning (Homogeneous)
   In Tuple, all values have different meaning (Heterogeneous)
'''
point_tuple = (5,9) # 5 is X_coordinate, 9 is Y_coordinate
point_list = [5,9] # both 5 and 9 are X_coordinates
In [68]:
# Second difference is tuples are immutable, meaning if we try to change we cannot, but we can change in a list

10. MODULES IN PYTHON¶

The whole idea of "reuse" applies well to the programming world. Module is a way to reuse a code written by someone else.

In [69]:
# Let's use math module
import math
math.sqrt(16)
Out[69]:
4.0
In [70]:
math.pow(2,5) # 2 is the number and 5 is the power to be mentioned
Out[70]:
32.0
In [71]:
# List of all the functions in math module. Using dir() command
dir(math)
Out[71]:
['__doc__',
 '__loader__',
 '__name__',
 '__package__',
 '__spec__',
 'acos',
 'acosh',
 'asin',
 'asinh',
 'atan',
 'atan2',
 'atanh',
 'ceil',
 'comb',
 'copysign',
 'cos',
 'cosh',
 'degrees',
 'dist',
 'e',
 'erf',
 'erfc',
 'exp',
 'expm1',
 'fabs',
 'factorial',
 'floor',
 'fmod',
 'frexp',
 'fsum',
 'gamma',
 'gcd',
 'hypot',
 'inf',
 'isclose',
 'isfinite',
 'isinf',
 'isnan',
 'isqrt',
 'lcm',
 'ldexp',
 'lgamma',
 'log',
 'log10',
 'log1p',
 'log2',
 'modf',
 'nan',
 'nextafter',
 'perm',
 'pi',
 'pow',
 'prod',
 'radians',
 'remainder',
 'sin',
 'sinh',
 'sqrt',
 'tan',
 'tanh',
 'tau',
 'trunc',
 'ulp']
In [72]:
math.log10(100)
Out[72]:
2.0
In [73]:
math.log10(1000)
Out[73]:
3.0
In [74]:
math.floor(2.3)
Out[74]:
2
In [75]:
math.ceil(2.3)
Out[75]:
3
In [77]:
# Let's use calendar module
import calendar
cal = calendar.month(2022,10)
print(cal)

    October 2022
Mo Tu We Th Fr Sa Su
                1  2
 3  4  5  6  7  8  9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31
In [78]:
calendar.isleap(2022)
Out[78]:
False
In [80]:
calendar.isleap(2016)
Out[80]:
True
In [81]:
# List of all functions in calendar module. Using dir()
dir(calendar)
Out[81]:
['Calendar',
 'EPOCH',
 'FRIDAY',
 'February',
 'HTMLCalendar',
 'IllegalMonthError',
 'IllegalWeekdayError',
 'January',
 'LocaleHTMLCalendar',
 'LocaleTextCalendar',
 'MONDAY',
 'SATURDAY',
 'SUNDAY',
 'THURSDAY',
 'TUESDAY',
 'TextCalendar',
 'WEDNESDAY',
 '_EPOCH_ORD',
 '__all__',
 '__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__spec__',
 '_colwidth',
 '_locale',
 '_localized_day',
 '_localized_month',
 '_monthlen',
 '_nextmonth',
 '_prevmonth',
 '_spacing',
 'c',
 'calendar',
 'datetime',
 'day_abbr',
 'day_name',
 'different_locale',
 'error',
 'firstweekday',
 'format',
 'formatstring',
 'isleap',
 'leapdays',
 'main',
 'mdays',
 'month',
 'month_abbr',
 'month_name',
 'monthcalendar',
 'monthrange',
 'prcal',
 'prmonth',
 'prweek',
 'repeat',
 'setfirstweekday',
 'sys',
 'timegm',
 'week',
 'weekday',
 'weekheader']
In [83]:
# We can also create our own module and use it in a program
# I have already created a module and I will just write the code in documentation 
'''
def calculate_triangle_area(base,height):
    return 1/2*(base*height)

def calculate_square_area(length):
    return length*length
'''
import functions
functions.calculate_square_area(5)
Out[83]:
25
In [85]:
# We can also use short forms for the imported modules
import functions as f
f.calculate_triangle_area(2,3)
# HEADS UP : To find a module, python will use current directory and then all directories listed in system path
Out[85]:
3.0
In [86]:
# To install a python module we use certain command for that
!pip install matplotlib
'''
For Jupyter Notebook I'm mentioning "!", in general you dont need to mention it.
And you need to have a internet connection to download and install a module

'''
# To uninstall use "pip uninstall module_name"

Requirement already satisfied: matplotlib in c:\anaconda3\lib\site-packages (3.5.3)
Requirement already satisfied: pillow>=6.2.0 in c:\anaconda3\lib\site-packages (from matplotlib) (9.2.0)
Requirement already satisfied: python-dateutil>=2.7 in c:\anaconda3\lib\site-packages (from matplotlib) (2.8.2)
Requirement already satisfied: cycler>=0.10 in c:\anaconda3\lib\site-packages (from matplotlib) (0.11.0)
Requirement already satisfied: packaging>=20.0 in c:\anaconda3\lib\site-packages (from matplotlib) (21.3)
Requirement already satisfied: fonttools>=4.22.0 in c:\anaconda3\lib\site-packages (from matplotlib) (4.33.3)
Requirement already satisfied: pyparsing>=2.2.1 in c:\anaconda3\lib\site-packages (from matplotlib) (3.0.9)
Requirement already satisfied: numpy>=1.17 in c:\anaconda3\lib\site-packages (from matplotlib) (1.22.3)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\anaconda3\lib\site-packages (from matplotlib) (1.4.2)
Requirement already satisfied: six>=1.5 in c:\anaconda3\lib\site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)
Out[86]:
'\nFor Jupyter Notebook I\'m mentioning "!", in general you dont need to mention it.\nAnd you need to have a internet connection to download and install a module\n\n'

11. WORKING WITH JSON IN PYTHON¶

WHAT IS JSON? - Java Script Object Notation. JSON is a data exchange format similar to XML.

In [89]:
'''
JSON(less volume of data)
----
{
    "name":"Siva",
    "address":"Pondicherry"
    "phone":"9876543210"
}

XML(more volume of data)
---
<name>Siva</name>
<address>Pondicherry</address>
<phone>9876543210</phone>

JSON is a lightweight format compared to XML

'''
# SAMPLE PROGRAM 1 - To create address book and write some records into it
book = {}
book['siva'] = {
    'name': 'siva',
    'address': 'Pondicherry',
    'phone': 9876543210
}
book['sankar'] = {
    'name': 'sankar',
    'address': 'Cuddalore',
    'phone': 9876543211
}

import json
s = json.dumps(book) # dumps() will take dictionary object book as an input then it is dumping it as its string. It will then convert to json format
with open("book.txt", "w") as f:
    f.write(s)

We can now read this JSON data using any language that supports JSON such as Javascript, C++ etc. Hence this is called data exchange format(i.e. exchanging data from python program to javascript program)

In [93]:
# Read the JSON records
f = open("book.txt","r")
s2 = f.read()
s2
Out[93]:
'{"siva": {"name": "siva", "address": "Pondicherry", "phone": 9876543210}, "sankar": {"name": "sankar", "address": "Cuddalore", "phone": 9876543211}}'
In [94]:
book2 = json.loads(s2) # "loads() is basically loading the string"
book2
Out[94]:
{'siva': {'name': 'siva', 'address': 'Pondicherry', 'phone': 9876543210},
 'sankar': {'name': 'sankar', 'address': 'Cuddalore', 'phone': 9876543211}}
In [95]:
book2['siva']
Out[95]:
{'name': 'siva', 'address': 'Pondicherry', 'phone': 9876543210}
In [96]:
book2['siva']['phone']
Out[96]:
9876543210
In [97]:
# To print all the records in the book2
for person in book2:
    print(book2[person])

{'name': 'siva', 'address': 'Pondicherry', 'phone': 9876543210}
{'name': 'sankar', 'address': 'Cuddalore', 'phone': 9876543211}

12. IMPORTANT PREDEFINED FUNCTION IN PYTHON¶

In [98]:
__name__
Out[98]:
'__main__'
In [99]:
'''You can see the above cell and it is that, it is entry point for any python program
    It is written as 
    if __name__ == "__main__"
Lets see an example

'''
def calculate_area(base, height):
    return 1/2*(base*height)

if __name__ == "__main__":
    a = calculate_area(10, 20)
    print("area: ",a)

area:  100.0
In [100]:
# area.py
def calculate_area(base, height):
    print("__name__: ",__name__)
    return 1/2*(base*height)
    
if __name__ == "__main__":
    print("I am in area.py")
    a = calculate_area(10, 20)
    print("area: ",a)

I am in area.py
__name__:  __main__
area:  100.0
In [103]:
# When does __name__ is something else other than main?
import area
# I created a module named area, I will show the code in documentation
'''
def calculate_area(base, height):
    print("__name__: ",__name__)
    return 1/2*(base*height)
    
if __name__ == "__main__":
    print("I am in area.py")
    a = calculate_area(10, 20)
    print("area: ",a)
'''
# You can show check the output on above cell
print("I am in caller.py")
area.calculate_area(5,10)
# You can see the output that __name__ is now area, because the entry point of this code is from the module "area"

I am in caller.py
__name__:  area
Out[103]:
25.0

13. EXCEPTION HANDLING IN PYTHON¶

WHAT ARE EXCEPTIONS? - Exceptions are errors detected at the time of program execution

In [104]:
'''
Let's say you're driving a car on a road going to some destination, and you reached it safely without any trouble. 
Road clear - Executing program without any exception
But life will not be same, some unexpected blockage is on the route, so you took a detour. From this you can say that
Blockage - Exception
Detour - Handling Exception

'''
1/0

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
Input In [104], in <cell line: 9>()
      1 '''
      2 Let's say you're driving a car on a road going to some destination, and you reached it safely without any trouble. 
      3 Road clear - Executing program without any exception
   (...)
      7 
      8 '''
----> 9 1/0

ZeroDivisionError: division by zero
In [106]:
# We got ZeroDivisionError on above cell
# This is called program crash
# Lets see how to handle a exception
# CASE 1 - Exception occurs
x = input("Enter number1: ")
y = input("Enter number2: ")
try:
    z = int(x) / int(y)
except ZeroDivisionError as e:
    print("Division by zero exception")
    z = None
print("Division is: ",z)

Enter number1: 1
Enter number2: 0
Division by zero exception
Division is:  None
In [107]:
# CASE 2 - Exception does not occurs
x = input("Enter number1: ")
y = input("Enter number2: ")
try:
    z = int(x) / int(y)
except Exception as e:
    print("Exception occured: ",e)
    z = None
print("Division is: ",z)

Enter number1: 6
Enter number2: 2
Division is:  3.0
In [110]:
# CASE 3 - Multiple Exception handled
x = input("Enter number1: ")
y = input("Enter number2: ")
try:
    z = x / int(y) # I did not mention the type which will throw error
except ZeroDivisionError as e:
    print("Division by zero exception")
    z = None
except TypeError as e:
    print('Type error exception')
    z = None
print("Division is: ",z)

Enter number1: 4
Enter number2: 2
Type error exception
Division is:  None

14. CLASSES AND OBJECT IN PYTHON¶

WHAT IS CLASS? - It applies not only to python but most of it since it falls under OOPS(Object Oriented Programming). A class is an abstraction of some entity which common sense of properties and matters.

WHAT IS OBJECT? - It is a specific instance of class. The behavior of class a varies. An object is created to work around that behavior.

In [112]:
# Class and object example
class Human:
    def __init__(self, name, occupation): # __init__ will initialize the properties of that particular class
        self.name = name # self keyword means the class itself its a mandatory syntax when you write class in python
        self.occupation = occupation
    
    def do_work(self):
        if self.occupation == "cricket player":
            print(self.name, "plays cricket")
        elif self.occupation == "cameraman":
            print(self.name, "take photos")
            
    def speaks(self):
        print(self.name, "says how are you?")
        
# creating instance 1
raj = Human("Rajesh Sharma","cameraman")
raj.do_work()
raj.speaks()

# creating instance 2
thala = Human("MS Dhoni","cricket player")
thala.do_work()
thala.speaks()

Rajesh Sharma take photos
Rajesh Sharma says how are you?
MS Dhoni plays cricket
MS Dhoni says how are you?

15. INHERITANCE IN PYTHON¶

In [115]:
# Lets see an example
class Vehicle:
    def general_usage(self):
        print("general use: transportation")

class Car(Vehicle): # Car inherits from Vehicle
    def __init__(self): # __init__ is seen as a constructor
        print("I am a car")
        self.wheels = 4
        self.has_roof = True
        
    def specific_usage(self):
        print("specific use: commute to work, vacation with family")
        
class MotorCycle(Vehicle): # MotorCycle inherits from Vehicle
    def __init__(self):
        print("I am a motor cycle")
        self.wheels = 2
        self.has_roof = False
        
    def specific_usage(self):
        print("specific use: road trip, racing")
        
# Creating instance for Car
c = Car()
c.general_usage() # This function comes from Vehicle class
c.specific_usage()

# Creating instance for MotorCycle
m = MotorCycle()
m.general_usage()
m.specific_usage()

'''
BENIFITS OF INHERITANCE:
-----------------------
1. Code Reuse
2. Extensibility
3. Readability

'''

I am a car
general use: transportation
specific use: commute to work, vacation with family
I am a motor cycle
general use: transportation
specific use: road trip, racing
In [118]:
# isinstance and issubclass methods
print(isinstance(c,Car)) # It will show True because c is an instance for Car
print(isinstance(c,MotorCycle)) # It will show False because c is not an instance for MotorCycle
print(issubclass(Car,Vehicle)) # It will show True since Car is subclass of Vehicle
print(issubclass(Car,MotorCycle)) # It will show False since Car is not a subclass of MotorCycle

True
False
True
False

16. MULTIPLE INHERITANCE IN PYTHON¶

In [120]:
# In this we will inherit a class from two different classes
class Father():
    def gardening(self):
        print("I do gardening")
    
class Mother():
    def cooking(self):
        print("I do cooking")
        
class Child(Father,Mother):
    def sports(self):
        print("I do sports")

# Creating Instance
c = Child()
c.gardening()
c.cooking()
c.sports()

I do gardening
I do cooking
I do sports

17. RAISE EXCEPTION AND FINALLY¶

In [123]:
# Raise Standard Exception
try:
    raise MemoryError('memory error') # MemoryError is built-in exception in python
except MemoryError as e:
    print(e)
# You can use both generic and specific exceptions
# NOTE : User defined exceptions are always derived from Exception base class

memory error
In [127]:
# Exception in a class
class Accident(Exception):
    def __init__(self,msg):
        self.msg = msg
    def handle(self):
        print("accident occured. take detour")
        
try:
    raise Accident('crash between two cars')
except Accident as e:
    e.handle()

accident occured. take detour

FINALLY STATEMENT - People use it to do cleanup, lets say we have a function, it opens a file and you have code block with potential of exception happening, but you have the exception unhandled if its an unknown, thats where we use finally statement, it will always execute code in finally block no matter what

In [129]:
# Lets see that in a program
def process_file():
    try:
        f = open("data.txt")
        x = 1/0
    except FileNotFoundError as e:
        print("inside except")
    finally:
        print("cleaning up file")
        f.close()
        
process_file() # It will throw exception, but still executes the finally block

cleaning up file

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
Input In [129], in <cell line: 12>()
      9         print("cleaning up file")
     10         f.close()
---> 12 process_file()

Input In [129], in process_file()
      3 try:
      4     f = open("data.txt")
----> 5     x = 1/0
      6 except FileNotFoundError as e:
      7     print("inside except")

ZeroDivisionError: division by zero

18. ITERATORS IN PYTHON:¶

What are iterators? - An object that contains a countable number of values

In [131]:
# Lets see an example
a = ["hey","bro","you'r","awesome"]
for i in a:
    print(i)
    
# Internally the loop goes through elements one by one using __iter__ built-in method

hey
bro
you'r
awesome
In [132]:
itr = iter(a)
itr
Out[132]:
<list_iterator at 0x27ac691d3d0>
In [133]:
next(itr)
Out[133]:
'hey'
In [134]:
next(itr)
Out[134]:
'bro'
In [135]:
next(itr)
Out[135]:
"you'r"
In [136]:
next(itr)
Out[136]:
'awesome'
In [137]:
next(itr) # Now it will throw an error that the iteration stopped

---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
Input In [137], in <cell line: 1>()
----> 1 next(itr)

StopIteration:
In [138]:
# Reverse iterator
itr = reversed(a) # It is also an built-in function
next(itr)
Out[138]:
'awesome'
In [139]:
next(itr)
Out[139]:
"you'r"
In [140]:
next(itr)
Out[140]:
'bro'
In [141]:
next(itr)
Out[141]:
'hey'

SAMPLE PROGRAM : Implement Remote Control Class that allows you to press "next" button to go to next TV channel

In [142]:
class RemoteControl():
    def __init__(self):
        self.channels = ["SunTV","VijayTV","KTV","Polimer"]
        self.index = -1 # I want to come to first to first channel, so while iterating -1+1 becomes 0 which is first index
        
    def __iter__(self):
        return self
    
    def __next__(self):
        self.index += 1
        if self.index == len(self.channels):
            raise StopIteration # You would have seen this on previous cells
            
        return self.channels[self.index] # it means channels[0].....channels[n]

# Creating Instance for our iterator
r = RemoteControl()
itr = iter(r)
print(next(itr))
print(next(itr))
print(next(itr))
print(next(itr))
# YOU CAN ALSO FOR LOOP TO PRINT THESE ITERATIONS WHILE PRINTING BUT ITS A TEDIOUS PROCESS TO KEEP IN MIND OF THE LENGTH AND NUMBER OF ITERATIONS

SunTV
VijayTV
KTV
Polimer

19. GENERATORS IN PYTHON¶

It is a simple way of creating iterators. These are the functions that return the traversal object.

In [143]:
def remote_control_next():
    yield 'VijayTV' # yield is used to preserve the state and resume execution when a successive function is called unlike return statement which completely destroys the state 
    yield 'KTV'
    
itr = remote_control_next()
itr # It will show that its a generator object
'''Using this will conserve memory since you are not calling all of the elements in one shot'''
Out[143]:
<generator object remote_control_next at 0x0000027AC74AC890>
In [144]:
next(itr)
Out[144]:
'VijayTV'
In [145]:
next(itr)
Out[145]:
'KTV'
In [146]:
for c in remote_control_next():
    print(c)

VijayTV
KTV
In [148]:
# SAMPLE PROGRAM - FIBONACCI SEQUENCE USING GENERATOR
def fib():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a+b # a becomes b and b becomes a+b
        
for f in fib():
    if f > 50: # keeping a limit since the looping should not go infinite
        break
    print(f)
    
'''
BENIFITS OF USING GENERATOR OVER CLASS BASED ITERATOR
1. You dont need to define iter() and next() methods
2. You dont need to raise StopIteration exception. It automatically raises it.
'''

0
1
1
2
3
5
8
13
21
34
Out[148]:
'\nBENIFITS OF USING GENERATOR OVER CLASS BASED ITERATOR\n1. You dont need to define iter() and next() methods\n2. You dont need to raise StopIteration exception. It automatically raises it.\n'

20. LIST, SET AND DICT COMPREHENSIONS¶

List/Set/Dict comprehensions provides a way to transform one list/set/dict into another

In [149]:
# LIST COMPREHENSION
# traditional way
numbers = [1,2,3,4,5,6,7]
even = []
for i in numbers:
    if i%2 == 0:
        even.append(i)
        
even
Out[149]:
[2, 4, 6]
In [151]:
# Using list comprehension
even = [i for i in numbers if i%2 ==0]
even

'''
CODE BREAKDOWN
--------------
even = [i(output variable) for i in numbers(for loop) if i%2 ==0(condition)]

'''
Out[151]:
[2, 4, 6]
In [153]:
# SQUARE OF NUMBERS
sqr_numbers = [i*i for i in numbers]
sqr_numbers
Out[153]:
[1, 4, 9, 16, 25, 36, 49]
In [154]:
# SET COMPREHENSION
# Set is an unordered collection of unique items
s = set([1,2,3,4,5,2,3])
s # It will clean the duplicates
Out[154]:
{1, 2, 3, 4, 5}
In [156]:
even = {i for i in s if i%2 == 0}
even
Out[156]:
{2, 4}
In [157]:
# DICT COMPREHENSIONS
# Two lists
cities = ["mumbai","newyork","paris"]
countries = ["india","usa","france"]
# Traditional way
z = zip(cities, countries) # zip() is used to zip two lists together
z
Out[157]:
<zip at 0x27ac7374840>
In [158]:
for a in z:
    print(a)

('mumbai', 'india')
('newyork', 'usa')
('paris', 'france')
In [160]:
# Using dict comprehension
d = {city:country for city, country in zip(cities,countries)}
d
Out[160]:
{'mumbai': 'india', 'newyork': 'usa', 'paris': 'france'}

21. SETS AND FROZEN SETS¶

A set is an unordered collection of unique elements

In [161]:
basket = {"orange","apple","mango","apple","orange"} 
type(basket)
Out[161]:
set
In [163]:
basket # It will remove duplicates
Out[163]:
{'apple', 'mango', 'orange'}
In [164]:
# Another way to initialize set
a = set()
a.add(1)
a.add(2)
a.add(3)
a
Out[164]:
{1, 2, 3}
In [165]:
# Since it is unordered we cannot do index operations
In [166]:
# We can use set to remove duplicates of a list but then the type changes
numbers = [1,2,3,4,2,3,4]
unique_numbers = set(numbers)
unique_numbers
Out[166]:
{1, 2, 3, 4}
In [167]:
# Frozen set - You cannot change content
fs = frozenset(numbers)
fs
Out[167]:
frozenset({1, 2, 3, 4})
In [168]:
# Frozenset does not allow to add new element, it will throw an error
fs.add(5)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [168], in <cell line: 2>()
      1 # Frozenset does not allow to add new element, it will throw an error
----> 2 fs.add(5)

AttributeError: 'frozenset' object has no attribute 'add'
In [169]:
# Basic operations in set
x = {"a","b","c"}

"a" in x
Out[169]:
True
In [170]:
"g" in x
Out[170]:
False
In [171]:
for i in x:
    print(i)

c
b
a
In [172]:
y = {"c", "d", "e"}
y
Out[172]:
{'c', 'd', 'e'}
In [173]:
# Sets has union and intersection operations
x | y # All elements without duplicates
Out[173]:
{'a', 'b', 'c', 'd', 'e'}
In [174]:
x & y # common elements between two sets
Out[174]:
{'c'}
In [175]:
x - y # It will subtract common elements
Out[175]:
{'a', 'b'}
In [176]:
x > y
Out[176]:
False
In [177]:
x < y
Out[177]:
False
In [181]:
x = {'d','e'}
x < y
Out[181]:
True

22. COMMAND LINE ARGUMENT PROCESSING USING ARGPARSE¶

code.png

If you run it on cmd, nothing happens, because it has nothing, it just initializing the parser and trying to pass the arguments

cmd.png

Now lets pass some real arguments. There are two kind of arguments,

  • Positional arguments
  • Optional arguments
In [ ]:
# Positional argument
'''
Here we are writing a program that makes 3 inputs,
1. First Number 
2. Second Number
3. Operation("add","subtract","multiply")

It should return result of operation based on inputs
'''

code1.png

cmd%200.png

In [183]:
# Now we will parse the arguments

cmd1.png

In [184]:
# Now we will write code for operations and execute it

code%202.png

cmd2.png

In [185]:
# Optional arguments
# To make argument optional we just add -- in front of argument name

code%203.png

In [186]:
# While executing using optional arguments, the order does not matter

cmd%203.png

In [187]:
# We can also skip arguments unlike positional arguments

cmd4.png

23. DECORATORS IN PYTHON¶

It is a design pattern in Python that allows a user to add new functionality to an existing object without modifying its structure

In [188]:
# Lets take a example how we decorate in traditional way
import time

def calc_square(numbers):
    start = time.time()
    result = []
    for number in numbers:
        result.append(number*number)
    end = time.time()
    print("calc_square took " + str((end-start)*1000) + " milliseconds")
    return result

def calc_cube(numbers):
    start = time.time()
    result = []
    for number in numbers:
        result.append(number*number*number)
    end = time.time()
    print("calc_cube took " + str((end-start)*1000) + " milliseconds")
    return result

array = range(1,100000)
out_square = calc_square(array)
out_cube = calc_cube(array)
    
'''
The problem with this code is that, lets say you have a complex software project and you have written 200 functions
so in order to measure performance of all those 200 functions, you have to write start and end time to each of it. 
And the actual functions logic and performance logic is combined in same function. It makes it less readable
'''.

calc_square took 7.998466491699219 milliseconds
calc_cube took 12.560129165649414 milliseconds
In [191]:
# We will use decorators which allow us to wrap the function in another function

def time_it(func):
    '''
    Functions are first class objects in python. What it means is that they can be treated just like
    any other variable and you can pass them as argument to another function or even return them as
    a return value
    '''
    def wrapper(*args, **kwargs): # *args - positional arguments, **kwargs - keyword arguments
        start = time.time()
        result = func(*args, **kwargs)
        end = time.time()
        print(func.__name__ + "took " + str((end-start)*1000) + " milliseconds")
        return result
    return wrapper

@time_it # way for using decorators
def calc_square(numbers):
    result = []
    for number in numbers:
        result.append(number*number)
    return result

@time_it
def calc_cube(numbers):
    result = []
    for number in numbers:
        result.append(number*number*number)
    return result

array = range(1,100000)
out_square = calc_square(array)
out_cube = calc_cube(array)

calc_squaretook 9.179353713989258 milliseconds
calc_cubetook 10.071277618408203 milliseconds

24. numpy library in python¶

It is used for working with arrays. It gives support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays.

In [192]:
import numpy as np
# One dimensional array
a = np.array([5,6,9])
a[0]
Out[192]:
5
In [193]:
a[1]
Out[193]:
6
In [194]:
a = np.array([[1,2],[3,4],[5,6]])
a.ndim # ndim is used to see dimensions
Out[194]:
2
In [195]:
a .itemsize
Out[195]:
4
In [196]:
# To change the datatype of the numpy array dtype argument is used
a = np.array([[1,2],[3,4],[5,6]], dtype=np.float64)
a.itemsize
Out[196]:
8
In [197]:
a.size # total number of elements
Out[197]:
6
In [199]:
a.shape # information on dimensions [rows and columns]
a
Out[199]:
array([[1., 2.],
       [3., 4.],
       [5., 6.]])
In [201]:
a = np.array([[1,2],[3,4],[5,6]], dtype=np.complex128)
a
Out[201]:
array([[1.+0.j, 2.+0.j],
       [3.+0.j, 4.+0.j],
       [5.+0.j, 6.+0.j]])
In [202]:
# Initializing array with placeholder numbers
np.zeros((3,4))
Out[202]:
array([[0., 0., 0., 0.],
       [0., 0., 0., 0.],
       [0., 0., 0., 0.]])
In [203]:
np.ones((3,4))
Out[203]:
array([[1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.]])
In [204]:
l = range(5)
l # creating list with range of 5
Out[204]:
range(0, 5)
In [205]:
l[0]
Out[205]:
0
In [206]:
l[1]
Out[206]:
1
In [207]:
np.arange(1,5)
Out[207]:
array([1, 2, 3, 4])
In [208]:
np.arange(1,5,2) # start,stop,step
Out[208]:
array([1, 3])
In [209]:
np.linspace(1,5,10) # it will generate 10 numbers with linearly spaced
Out[209]:
array([1.        , 1.44444444, 1.88888889, 2.33333333, 2.77777778,
       3.22222222, 3.66666667, 4.11111111, 4.55555556, 5.        ])
In [210]:
np.linspace(1,5,5)
Out[210]:
array([1., 2., 3., 4., 5.])
In [213]:
print(a.shape)
a

(3, 2)
Out[213]:
array([[1.+0.j, 2.+0.j],
       [3.+0.j, 4.+0.j],
       [5.+0.j, 6.+0.j]])
In [212]:
a.reshape(2,3)
Out[212]:
array([[1.+0.j, 2.+0.j, 3.+0.j],
       [4.+0.j, 5.+0.j, 6.+0.j]])
In [214]:
a.reshape(6,1)
Out[214]:
array([[1.+0.j],
       [2.+0.j],
       [3.+0.j],
       [4.+0.j],
       [5.+0.j],
       [6.+0.j]])
In [216]:
print(a.ravel()) # to make it flat and make it 1D. It does not touch original array
a

[1.+0.j 2.+0.j 3.+0.j 4.+0.j 5.+0.j 6.+0.j]
Out[216]:
array([[1.+0.j, 2.+0.j],
       [3.+0.j, 4.+0.j],
       [5.+0.j, 6.+0.j]])
In [217]:
a.min()
Out[217]:
(1+0j)
In [218]:
a.max()
Out[218]:
(6+0j)
In [219]:
a.sum()
Out[219]:
(21+0j)
In [220]:
a.sum(axis=0) # vertical axis 0, horizontal axis 1
Out[220]:
array([ 9.+0.j, 12.+0.j])
In [221]:
a.sum(axis=1)
Out[221]:
array([ 3.+0.j,  7.+0.j, 11.+0.j])
In [222]:
np.sqrt(a) # compute square root of each element
Out[222]:
array([[1.        +0.j, 1.41421356+0.j],
       [1.73205081+0.j, 2.        +0.j],
       [2.23606798+0.j, 2.44948974+0.j]])
In [223]:
np.std(a) # compute standard deviation
Out[223]:
1.707825127659933
In [225]:
a = np.array([[1,2],[3,4]])
In [226]:
b = np.array([[5,6],[7,8]])
In [227]:
a
Out[227]:
array([[1, 2],
       [3, 4]])
In [228]:
b
Out[228]:
array([[5, 6],
       [7, 8]])
In [229]:
a + b
Out[229]:
array([[ 6,  8],
       [10, 12]])
In [230]:
a - b
Out[230]:
array([[-4, -4],
       [-4, -4]])
In [231]:
a * b
Out[231]:
array([[ 5, 12],
       [21, 32]])
In [232]:
a / b
Out[232]:
array([[0.2       , 0.33333333],
       [0.42857143, 0.5       ]])
In [233]:
a.dot(b) # matrix product
Out[233]:
array([[19, 22],
       [43, 50]])
In [234]:
'''
3 main benifits of numpy array over a python list,
1. Less Memory
2. Fast
3. Convinient

'''
# lets compare how it has less memory
import numpy as np
import sys

l = range(1000)
print(sys.getsizeof(5)*len(l))

array = np.arange(1000)
print(array.size*array.itemsize)

# From the output we can see that List took 28 bytes of memory size while numpy array took only 4 bytes

28000
4000
In [236]:
# Now lets see how numpy array is fast and convinient
import time

SIZE = 10000000

l1 = range(SIZE)
l2 = range(SIZE)

a1 = np.arange(SIZE)
a2 = np.arange(SIZE)

start = time.time()
result = [(x+y) for x,y in zip(l1,l2)]
print("python list took: ",(time.time()-start)*1000)

start = time.time()
result = a1 + a2 # it is convinient, does not need to use list comprehension
print("numpy took: ",(time.time()-start)*1000)

# list took 773 milliseconds while numpy array only took 72 milliseconds -_-

python list took:  773.7343311309814
numpy took:  72.00193405151367
In [237]:
# Indexing and slicing
n = [6,7,8]
n[0:2] # slicing in normal list
Out[237]:
[6, 7]
In [239]:
c = np.array([6,7,8])
c[0:2] # same concept can be used in numpy arrays also
Out[239]:
array([6, 7])
In [240]:
a
Out[240]:
array([[1, 2],
       [3, 4]])
In [242]:
a = np.array([[6,7,8],[1,2,3],[9,3,2]])
In [243]:
a[1,2]
Out[243]:
3
In [244]:
a[0:2, 2] # 0 to 1st row and then to second element
Out[244]:
array([8, 3])
In [245]:
for row in a:
    print(row)

[6 7 8]
[1 2 3]
[9 3 2]
In [246]:
for cell in a.flat: # flatten it as 1D array
    print(cell)

6
7
8
1
2
3
9
3
2
In [248]:
a = np.arange(6).reshape(3,2)
b = np.arange(6,12).reshape(3,2)
In [249]:
a
Out[249]:
array([[0, 1],
       [2, 3],
       [4, 5]])
In [250]:
b
Out[250]:
array([[ 6,  7],
       [ 8,  9],
       [10, 11]])
In [251]:
np.vstack((a,b)) # we can stack two arrays vertically
Out[251]:
array([[ 0,  1],
       [ 2,  3],
       [ 4,  5],
       [ 6,  7],
       [ 8,  9],
       [10, 11]])
In [252]:
np.hstack((a,b)) # we can stack two arrays horizontally
Out[252]:
array([[ 0,  1,  6,  7],
       [ 2,  3,  8,  9],
       [ 4,  5, 10, 11]])
In [255]:
a = np.arange(30).reshape(2,15)
a
Out[255]:
array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]])
In [256]:
r = np.hsplit(a,3) # we are splitting horizontally into 3
r[0]
Out[256]:
array([[ 0,  1,  2,  3,  4],
       [15, 16, 17, 18, 19]])
In [257]:
r[1]
Out[257]:
array([[ 5,  6,  7,  8,  9],
       [20, 21, 22, 23, 24]])
In [258]:
r[2]
Out[258]:
array([[10, 11, 12, 13, 14],
       [25, 26, 27, 28, 29]])
In [259]:
v = np.vsplit(a,2)
v[0]
Out[259]:
array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14]])
In [260]:
v[1]
Out[260]:
array([[15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]])
In [261]:
a = np.arange(12).reshape(3,4)
a
Out[261]:
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
In [264]:
b = a > 4 # its gonna store the result in array form
In [265]:
b
Out[265]:
array([[False, False, False, False],
       [False,  True,  True,  True],
       [ True,  True,  True,  True]])
In [267]:
a[b] # index of an array is array itself, wherever it found true it return only those values
Out[267]:
array([ 5,  6,  7,  8,  9, 10, 11])
In [270]:
a[b] = -1 # any element greater than 4 it replaced as -1
a
Out[270]:
array([[ 0,  1,  2,  3],
       [ 4, -1, -1, -1],
       [-1, -1, -1, -1]])
In [271]:
# iterating numpy arrays
a = np.arange(12).reshape(3,4)
a
Out[271]:
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
In [273]:
for cell in a.flatten():
        print(cell)

0
1
2
3
4
5
6
7
8
9
10
11
In [275]:
# nditer - Efficient multi dimensional iterator object to iterate over arrays
for x in np.nditer(a, order='F'): # fortran order
    print(x)

0
4
8
1
5
9
2
6
10
3
7
11
In [276]:
# so many properties are given in original python documentation
# python is not about knowing every single thing
# Its about you knowing where to check for what

25. READING AND WRITING FILES¶

In [279]:
'''
We will look into,
1. Create a new file and write to it
2. Reading file line by line
3. File open modes
4. With statement

'''
f = open("sample.txt","w") # r , w , r+ , w+ , a  
f.write("Howdy mates")
f.close()

# to append a line to a file
f = open("sample.txt","a")
f.write("\nHow are you doing?")
f.close()

samplef.png

In [280]:
# I have created a file called dbms and saved it to directory
# We will read the file line by line and aslo count the number of words in each line and append the word count to end of line

image.png

In [281]:
f = open("dbms.txt","r")
print(f.read())
f.close()

Differentiate between 
1NF
2NF
3NF
BCNF
4NF
5NF
(Write any seven points)
In [291]:
# Now appending the count
f = open("dbms.txt","r")
f_out = open("sample_out.txt","w") # creating output file
for line in f:
    tokens = line.split(' ')
    # split() will split the string using input character and return a list (array) of words
    f_out.write("word count:"+str(len(tokens)) + " " + line)
f.close()
f_out.close()

'''
FILE OPEN MODES:
---------------
r - Opens file for reading only. Throws error if file does not exist
w - Opens file for writing only. If file doesnt exist then it will create new one. If exist then it will overwrite it
r+ - Opens file for both reading and writing
w+ - Opens file for reading and writing. If file doesnt exist then it will create new one. If it exist then it will 
     overwrite it.
a - Opens a file in append mode. Whatever you write to file will get appended and original content will not be overwritten

'''

image.png

In [292]:
# with statement
# we dont need explicitly close the file everytime
# i.e. You dont need to do f.close() if you use with.
# Using with will automatically close the file for you
with open("sample.txt","r") as f:
    print(f.read())
    
print(f.closed) # f.closed flag tells us if the file is closed or still open
# This is just to show the working of with, not mandatory to use f.closed

Howdy mates
How are you doing?
True

26. MULTITHREADING IN PYTHON¶

It enables CPUs to run different parts(threads) of a process concurrently to maximize CPU utilization

In [303]:
'''
SAMPLE PROGRAM
--------------
For a given list of numbers print square and cube of every numbers
For example,
    Input : [2,3,8,9]
    Output : Square list - [4,9,64,81]
             Cube list - [8,27,512,729]
        
'''
# Lets frame it in a program in traditional way
import time

def calc_square(numbers):
    print("Calculate square numbers")
    for n in numbers:
        time.sleep(0.2) # keeping CPU idle
        print('square: ',n*n)
        
def calc_cube(numbers):
    print("Calculate cube numbers")
    for n in numbers:
        time.sleep(0.2)
        print('cube: ',n*n*n)
        
arr = [2,3,8,9]

t = time.time()
calc_square(arr)
calc_cube(arr)

print("done in: ",time.time()-t)
print("Hah... I am done with all my work now!")

Calculate square numbers
square:  4
square:  9
square:  64
square:  81
Calculate cube numbers
cube:  8
cube:  27
cube:  512
cube:  729
done in:  1.6621217727661133
Hah... I am done with all my work now!
In [304]:
# Using multithreading we can decrease amount of time it takes
import time
import threading

def calc_square(numbers):
    print("Calculate square numbers")
    for n in numbers:
        time.sleep(0.2) # keeping CPU idle
        # This is an artificial delay I made, there some cases like web services, where delay happens
        print('square: ',n*n)
        
def calc_cube(numbers):
    print("Calculate cube numbers")
    for n in numbers:
        time.sleep(0.2)
        print('cube: ',n*n*n)
        
arr = [2,3,8,9]

t = time.time()

t1 = threading.Thread(target=calc_square, args=(arr,)) # We can also parse multiple arguments
t2 = threading.Thread(target=calc_cube, args=(arr,))

t1.start()
t2.start()

t1.join() # join() will make the thread to wait until another thread does its job 
t2.join() 

print("done in: ",time.time()-t)
print("Hah... I am done with all my work now!")

# You can compare both traditional way and using multithreading
# It gets decreased from 1.6s ---> 0.8s, its literally decreased half the time

Calculate square numbersCalculate cube numbers

square:  4
cube:  8
square:  9
cube:  27
square:  64
cube:  512
square:  81
cube:  729
done in:  0.8565447330474854
Hah... I am done with all my work now!

27. MULTIPROCESSING IN PYTHON¶

It is a package that supports spawning processes using an API similar to the threading module.

In [311]:
'''
Lets create two processes,
1. First is to calculate square of all numbers
2. Second one is to calculate cube of numbers
 
Note: Functionality within this package requires that the __main__ method be importable by the children. 
This is covered in Programming guidelines however it is worth pointing out here. 
This means that some examples, such as the multiprocessing.Pool examples will not work in the interactive interpreter.
So I will vs code and share screenshot and the code is in the documentation


import multiprocessing

square_result = []
cube_result = []

def calc_square_ml(numbers):
    global square_result # it recognized as global variable
    for n in numbers:
        print('square: ' + str(n*n))
        square_result.append(n*n)
    print('within a process: result ' + str(square_result))
        
def calc_cube_ml(numbers):
    global cube_result
    for n in numbers:
        print('cube: ' + str(n*n*n))
        cube_result.append(n*n*n)
    print('within a process: result ' + str(cube_result))
    
if __name__ == "__main__":
    arr = [2,3,8,9]
    p1 = multiprocessing.Process(target=calc_square_ml, args = (arr,))
    p2 = multiprocessing.Process(target=calc_cube_ml, args = (arr,))
    
    p1.start()
    p2.start()
    
    p1.join()
    p2.join()
    
    print("Done!")
    
'''
print("CHECK ORIGINAL PYTHON DOCUMENTATION FOR MORE DETAILS")

CHECK ORIGINAL PYTHON DOCUMENTATION FOR MORE DETAILS

image.png

28. SHARING DATA BETWEEN PROCESSES: VALUE AND ARRAY¶

In [312]:
'''
import multiprocessing

result = []

def calc_square(numbers):
    global result
    for n in numbers:
        result.append(n*n)
    print('inside process ' + str(result))

if __name__ == "__main__":
    numbers = [2,3,5]
    p = multiprocessing.Process(target=calc_square, args=(numbers,))

    p.start()
    p.join()

    print('outside process ' + str(result))
    
    
If we see this code, it gives output like inside process has the values but not the outside process,
Lets see why, whenever a new process is created, the new process get its own address space, a address space is 
the place where it stores all variables etc. 

'''
print("Look at below image")

Look at below image

image.png

As in the diagram, left side is the global variable memory present in main, while its copy is in child program, this is a problem, since it cant be changed as it is.

But there is way to share memory which lives outside process image.png

In [313]:
'''
There are two ways to share memory in multiprocessing,
1. Using array
2. Using value

'''
print("Look at the image")

Look at the image
In [314]:
# Using Array
'''
import multiprocessing

def calc_square(numbers):
    for idx, n in enumerate(numbers):
        result[idx] = n*n

if __name__ == "__main__":
    numbers = [2,3,5]
    result = multiprocessing.Array('i,3') # 'd' means double, 'i' means integer
    p = multiprocessing.Process(target=calc_square, args=(numbers,))

    p.start()
    p.join()

    print(" Result: ",result[:]) # ":" - way to print all elements in array
    
'''
print("Look below for vs code and output")

Look below for vs code and output

image.png

In [315]:
# Using Value
'''
import multiprocessing

def calc_square(numbers, v):
    v.value = 5.67

if __name__ == "__main__":
    numbers = [2,3,5]
    v = multiprocessing.Value('d', 0.0)
    p = multiprocessing.Process(target=calc_square, args=(numbers, v))

    p.start()
    p.join()

    print(v.value)

child process is updating values and still parent process able to access it

'''
print("Look below cell")

Look below cell

image.png

29. SHARING DATA BETWEEN PROCESSES USING QUEUE¶

image.png

P1 and P2 has own address space whenever they need to share data they use shared memory. Queue basically works on shared memory. So lets see how it used in program

In [316]:
'''
import multiprocessing

def calc_square(numbers, q):
    for n in numbers:
        q.put(n*n) # FIFO concept - insert data at end and pull from end

if __name__ == "__main__":
    numbers = [2,3,5]
    q = multiprocessing.Queue()
    p = multiprocessing.Process(target=calc_square, args=(numbers, q))

    p.start()
    p.join()

    while q.empty() is False:
        print(q.get())
        
        
Queue is used to share data within two processes(parent, child).

D/B Queue module and multiprocessing queue module
-------------------------------------------

-----------------------------------------|-------------------------------------------|
    Multiprocessing Queue                |            Queue                          |
-----------------------------------------|-------------------------------------------|
import multiprocessing                   |        import queue                       |
q = multiprocessing.Queue()              |        q = queue.Queue()                  |
-----------------------------------------|-------------------------------------------|                                         
Lives in shared memory                   |        Lives in in-process memory         |
-----------------------------------------|-------------------------------------------|                                     
Used to share data between processes     |        Used to share data between threads |
-----------------------------------------|-------------------------------------------|

'''

print("Look below for vs code output")

Look below for vs code output

image.png

30. MULTIPROCESSING LOCK IN PYTHON¶

In [317]:
# In real life why we need lock?
# There are some resources where two people cant use it at the same time, like Bathroom.
# It would be pretty embarassing if two people use it at the same time.
# So in programming world, when two process try access shared resource, it can create problem, we need to protect it with lock

image.png If we look at the code and output, It prints 200 on first run, and it printed 185 on second run. It is because when one process runs, another processes also runs using the same resource which makes inappropriate and inconsistent results. This is why we use locks

In [318]:
'''
CODE WITH LOCK APPLIED
----------------------

import time
import multiprocessing

def deposit(balance, lock):
    for i in range(100):
        time.sleep(0.1)
        lock.acquire()
        balance.value = balance.value + 1 
        lock.release()
        # we put acquire and release only on critical section which is the logic of the code

def withdraw(balance, lock):
    for i in range(100):
        time.sleep(0.1)
        lock.acquire()
        balance.value = balance.value - 1
        lock.release()

if __name__ == '__main__':
    balance = multiprocessing.Value('i', 200)
    lock = multiprocessing.Lock()
    d = multiprocessing.Process(target=deposit,args=(balance,lock))
    w = multiprocessing.Process(target=withdraw,args=(balance,lock))
    d.start()
    w.start()
    d.join()
    w.join()
    print(balance.value)
    
'''

print("Look below for output")

Look below for output

image.png So now, if you see, after using lock, it will give out same result, even if you run it multiple times

31. MULTIPROCESSING POOL IN PYTHON¶

In [319]:
# If we see the below program there's nothing wrong in there. But we'll how it internally works

image.png The cores are processing unit, here we have 4 cores, when we run a program the OS gonna select one core. But other cores are sitting idle. But if we do much more complex and heavy, then it makes it difficult for one core to handle it, so we will divide the work.

PARALLEL PROCESSING image-2.png

In [320]:
# lETS SEE A CODE EXAMPLE
'''
from multiprocessing import Pool

def f(n):
    return n*n
if __name__ == "__main__":
    array = [1,2,3,4,5]
    p = Pool()
    result = p.map(f,array) # It will divide the work to respective cores
    print(result)

'''

print("Look at the vs code and output")

Look at the vs code and output

image.png Visually it has no difference, but internally it did divide the work, Lets see the time performance to get an understanding

In [321]:
'''
from multiprocessing import Pool
import time

def f(n):
    sum = 0
    for x in range(1000):
        sum += x*x
    return sum

if __name__ == "__main__":
    
    t1 = time.time()
    p = Pool()
    result = p.map(f,range(100000)) # It will divide the work to respective cores
    p.close()
    p.join()

    print("Pool took:",time.time()-t1)

    t2 = time.time()
    result = []
    for x in range(100000):
        result.append(f(x))

    print("Serial processing took:",time.time()-t2)
    


'''

print("Look at the vs code and output")

Look at the vs code and output

image.png Pool only took 5s, where normal serial processing took 16s.

32. PYTHON UNIT TESTING - PYTEST¶

PYTHON TESTING FRAMEWORKS

  • UNIT TEST
  • NOSE
  • PYTEST

Among these 3, pytest is the best

In [324]:
# Lets see an example
'''
I already created a module named math_t.py. Mentioned the code in documentation

def calc_total(a,b):
    return a+b
def calc_multiply(a,b):
    return a*b
'''
print("After creating module, we need to create a unit test module")

After creating module, we need to create a unit test module
In [323]:
# test_math.py
import math_t

def test_calc_total():
    total = math_t.calc_total(4,5)
    assert total == 9
    
def test_calc_multiply():
    result = math_t.calc_multiply(10,3)
    assert result == 30
In [325]:
# There are two ways to do unit testing in python
# First one is using command "python -m pytest", It will search for files that has prefix "test_" and run the tests for it

image.png

In [326]:
# Second one is using command "py.test"

image.png

PYTEST: PARAMETERIZED TESTS We can combine multiple test cases into a single test case

In [327]:
# In normal unit tests, if we want to write multiple tests, we will write it like this
'''
import math_t

def test_calc_square_1():
    total = math_t.calc_square(4)
    assert total == 16
    
def test_calc_square_2():
    result = math_t.calc_square(3)
    assert result == 9

def test_calc_square_3():
    result = math_t.calc_square(6)
    assert result == 36
    
But there is a problem with this, we cant write 100's of test cases like this, so thats where parameterized tests 
comes into the play

'''
print("Lets look further")

Lets look further
In [ ]:
# We can write it like this

import math_t
import pytest

@pytest.mark.parametrize("test_input, expected_output", # We used pytest decorator and passed parameters - i/p, o/p, tuple
                         [
                            (4, 16),
                            (3, 9),
                            (6, 36)
                         ]
                         )
def test_calc_square_1(test_input, expected_output):
    total = math_t.calc_square(test_input)
    assert total == expected_output

image.png

In [1]:
# AND THAT'S THE END OF IT
# STILL PYTHON HAS LOT OF LIBRARIES AND MICRO MODULES 
# PYTHON IS ALL ABOUT IMPLYING IT ON SOMETHING. DO THAT RIGHT, ITS GONNA BE AN EASY JOURNEY
In [ ]: